Build queue for --watch #1189

james-allan-lloyd · 2021-09-30T06:59:56Z

Use a queue for build requests in --watch mode so that duplicate builds can be removed, and optionally use a solution to build multiple projects to avoid dependencies being built simultaneously and causing file locking races.

Example config (albeit the simplest approach):

name: tye-build-fix
solution: tye-build-fix.sln
services:
- name: frontend
  project: frontend/frontend.csproj
- name: backend
  project: backend/backend.csproj
# both projects depend on SharedLibrary/SharedLibrary.csproj

note-1: I'm new to C# but experienced in other languages. It's very possible that I am not doing things in a canonical C# way.
note-2: to work around a bug in msbuild when using the -target commandline arg, I had to upgrade msbuild past version 16.10 (I'm using 16.11.0.36601 specifically).

Resolves #580
Resolves #1187

Also needed to update msbuild version past 16.10 because of a bug that meant that the default targets of projects couldn't be built.

dnfadmin · 2021-09-30T07:00:20Z

All CLA requirements met.

dbreshears · 2021-10-06T19:20:29Z

This resolves #1187

davidfowl · 2021-10-10T17:39:36Z

src/Microsoft.Tye.Hosting/WatchBuilderWorker.cs

+                    var solutionBatch = new Dictionary<string, List<BuildRequest>>(); //  store the list of promises
+                    var projectBatch = new Dictionary<string, List<BuildRequest>>();
+                    // TODO: quiet time... maybe wait both...?
+                    await Task.Delay(100);


Why is there a delay here?

I had the idea of having a quiet time where we wait to see if more files will be saved, then schedule the builds at once... it would be better to have it waiting 100ms (or a more sensible duration) after the last WaitToReadAsync(). Unsure what would happen if it was removed; might still need to yield to let duplicate project build requests enter the queue?

philliphoff · 2022-02-03T19:07:41Z

@james-allan-lloyd Sorry for it taking so long to be able to get to this. I hope you don't take offense to the changes I've made; feel free to suggest others. I'd appreciate if you could take the latest code for a spin and see if it continues to work in the way you expect.

philliphoff · 2022-02-03T21:54:34Z

@ravipal This is another one to review, when you have a chance. (Comment added since the PR's been open a while and the original notification would've been long ago.)

james-allan-lloyd · 2022-02-04T07:04:25Z

@james-allan-lloyd Sorry for it taking so long to be able to get to this. I hope you don't take offense to the changes I've made; feel free to suggest others. I'd appreciate if you could take the latest code for a spin and see if it continues to work in the way you expect.

No worries, I'll get back to you soon on it.

src/Microsoft.Tye.Hosting/BuildWatcher.cs

docs/reference/schema.md

src/schema/tye-schema.json

philliphoff · 2022-02-09T22:25:27Z

Moving forward with this PR in order to make progress toward 0.11; we'll deal with any fallout, if any, as separate PRs.

james-allan-lloyd · 2022-02-10T10:45:44Z

Hi guys, I only just now got to checking out the changes and they work for me (4 or so projects depending on a 2 shared libraries).

One small caveat; when using the solution build, it doesn't like nested targets (I'm using dotnet msbuild version 17.0.0+c9eb9dd64). This appears to be the default now when doing dotnet sln add... I worked around it by commenting out the GlobalSection(NestedProjects) section in the solution file, but maybe there's a way to get the nested name from the file?

To be clear, my day to day work project looks a bit like this:

api-services
- serviceA
  - serviceA.csproj
- serviceB
  - serviceB.csproj
tye.yaml
MySolution.sln

I need to invoke msbuild like this to build a targeted project:

dotnet msbuild -target:api-services\serviceA

a-shoemaker · 2022-08-17T17:44:24Z

src/Microsoft.Tye.Hosting/BuildWatcher.cs

+                                {
+                                    logger.LogInformation("Build Watcher: Building {Targets} of solution {SolutionPath}...", targets, solutionPath);
+
+                                    var buildResult = await ProcessUtil.RunAsync("dotnet", $"msbuild {solutionPath} -target:{targets}", throwOnError: false, workingDirectory: workingDirectory, cancellationToken: cancellationToken);


Note for #1438, this code is not replacing "special characters" with an _ in the targets.

https://docs.microsoft.com/en-us/visualstudio/msbuild/how-to-build-specific-targets-in-solutions-by-using-msbuild-exe?view=vs-2022
(I believe that documentation has a bit of a bug in itself, that last "in the specified target name" should be "in the specified project name" to not be confused with the target name portion of -targets argument.)

James Lloyd added 4 commits September 30, 2021 08:45

Central build queue, w/ build from solution

cebf823

Fix some warnings

06bdc5d

Need to use -target and not -targets

5c47ab4

Also needed to update msbuild version past 16.10 because of a bug that meant that the default targets of projects couldn't be built.

Remove unnecessary function

7f518b5

james-allan-lloyd requested review from philliphoff and pratiksanglikar as code owners September 30, 2021 06:59

dbreshears mentioned this pull request Oct 6, 2021

Use a build queue to avoid file locking and optimise build time #1187

Closed

dbreshears removed the request for review from pratiksanglikar October 6, 2021 19:18

philliphoff self-assigned this Oct 8, 2021

davidfowl reviewed Oct 10, 2021

View reviewed changes

Handle failures building solution better

298a24c

james-allan-lloyd requested a review from ravipal as a code owner October 14, 2021 05:54

james-allan-lloyd and others added 13 commits October 14, 2021 07:55

Merge branch 'dotnet:main' into build-queue

6835a9b

Merge branch 'main' into build-queue

55129eb

Styling updates.

1616660

Scaffold explicit start/stop of builds.

3736765

More styling updates.

e825a0e

Cleanup working directory.

8ff450c

Make build result more consistent.

181a239

Tweak logging.

b562362

Update formatting.

1711ea5

Update more formatting.

85440d5

Add locks for critical areas.

bb1ae72

Update schema and reference doc.

b901e2d

Fix formatting.

3acd282

ravipal approved these changes Feb 4, 2022

View reviewed changes

samtrion reviewed Feb 7, 2022

View reviewed changes

src/Microsoft.Tye.Hosting/BuildWatcher.cs Outdated Show resolved Hide resolved

Updates per PR feedback.

5a7b5cd

samtrion reviewed Feb 8, 2022

View reviewed changes

docs/reference/schema.md Outdated Show resolved Hide resolved

src/schema/tye-schema.json Outdated Show resolved Hide resolved

More updates per PR feedback.

931558a

philliphoff merged commit f677a15 into dotnet:main Feb 9, 2022

a-shoemaker reviewed Aug 17, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build queue for --watch #1189

Build queue for --watch #1189

james-allan-lloyd commented Sep 30, 2021 •

edited by dbreshears

Loading

dnfadmin commented Sep 30, 2021 •

edited

Loading

dbreshears commented Oct 6, 2021

davidfowl Oct 10, 2021

james-allan-lloyd Oct 14, 2021

philliphoff commented Feb 3, 2022

philliphoff commented Feb 3, 2022

james-allan-lloyd commented Feb 4, 2022

philliphoff commented Feb 9, 2022

james-allan-lloyd commented Feb 10, 2022

a-shoemaker Aug 17, 2022 •

edited

Loading

Build queue for --watch #1189

Build queue for --watch #1189

Conversation

james-allan-lloyd commented Sep 30, 2021 • edited by dbreshears Loading

dnfadmin commented Sep 30, 2021 • edited Loading

dbreshears commented Oct 6, 2021

davidfowl Oct 10, 2021

Choose a reason for hiding this comment

james-allan-lloyd Oct 14, 2021

Choose a reason for hiding this comment

philliphoff commented Feb 3, 2022

philliphoff commented Feb 3, 2022

james-allan-lloyd commented Feb 4, 2022

philliphoff commented Feb 9, 2022

james-allan-lloyd commented Feb 10, 2022

a-shoemaker Aug 17, 2022 • edited Loading

Choose a reason for hiding this comment

james-allan-lloyd commented Sep 30, 2021 •

edited by dbreshears

Loading

dnfadmin commented Sep 30, 2021 •

edited

Loading

a-shoemaker Aug 17, 2022 •

edited

Loading